An Unsupervised Sampling Approach for Image-Sentence Matching Using Document-level Structural Information

In this paper, we focus on the problem of unsupervised image-sentence matching. Existing research explores to utilize document-level structural information sample positive and negative instances for model training. Although approach achieves results, it introduces a sampling bias fails distinguish with high semantic similarity. To alleviate bias, propose new strategy select additional intra-document pairs as or samples. Furthermore, recognize complex pattern in samples, Transformer based capture fine-grained features implicitly construct graph each document, where concepts document are introduced bridge representation learning images sentences context document. Experimental results show effectiveness our learn well-aligned multimodal representations.